<html>
<head>
<title>ZFS Message ID: ZFS-8000-ER</title>
</head>
<body>
<!--
 CDDL HEADER START

 The contents of this file are subject to the terms of the
 Common Development and Distribution License (the "License").
 You may not use this file except in compliance with the License.

 You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
 or http://www.opensolaris.org/os/licensing.
 See the License for the specific language governing permissions
 and limitations under the License.

 When distributing Covered Code, include this CDDL HEADER in each
 file and include the License file at usr/src/OPENSOLARIS.LICENSE.
 If applicable, add the following below this CDDL HEADER, with the
 fields enclosed by brackets "[]" replaced with your own identifying
 information: Portions Copyright [yyyy] [name of copyright owner]

 CDDL HEADER END
-->
<!-- start: templates/success -->
<!-- ident      "@(#)success    1.4     02/14/14 SMI" -->
Article for Message ID:&nbsp;&nbsp; <font size="+1"><tt><b>ZFS-8000-ER</b></tt></font>
<br><br>
<hr>
<br><br>
<!-- end: templates/success -->
<!-- start: article -->
<!-- code: ZFS-8000-ER -->
<!-- keys: errata -->
<table width="80%"><tr><td>
<a name="1"><b>ZFS Errata #1</b></a>
<p>
<dl>
<dt><p><b>Type</b>
<dd>Compatibility
<dt><p><b>Severity</b>
<dd>Moderate
<dt><p><b>Description</b>
<dd>The ZFS pool contains an on-disk format incompatibility.
<dt><p><b>Automated Response</b>
<dd>No automated response will be taken.
<dt><p><b>Impact</b>
<dd>Until the pool is scrubbed using ZoL version 0.6.3 or newer the pool may not be imported by older versions of ZoL or other non-Linux ZFS implementations.
<dt><p><b>Suggested Action for System Administrator</b>
<dd>
The pool contains an on-disk format incompatibility.  Affected pools must be
imported and scrubbed using the current version of ZFS.  This will return the
pool to a state in which it may be imported by other implementations.  This
errata only impacts compatibility between ZFS versions, no user data is at
risk as result of this erratum.

<pre>
# zpool status -x
  pool: test
 state: ONLINE
status: Errata #1 detected.
action: To correct the issue run 'zpool scrub'.
   see: https://zfsonlinux.org/msg/ZFS-8000-ER
  scan: none requested
config:

	NAME            STATE     READ WRITE CKSUM
	test            ONLINE    0    0     0
	  raidz1-0      ONLINE    0    0     0
	    vdev0       ONLINE    0    0     0
	    vdev1       ONLINE    0    0     0
	    vdev2       ONLINE    0    0     0
	    vdev3       ONLINE    0    0     0

errors: No known data errors

# zpool scrub test

# zpool status -x
all pools are healthy
</pre>
</td></tr></table>
<br><br>
<hr>
<br><br>
<!-- end: templates/success -->
<!-- start: article -->
<!-- code: ZFS-8000-ER -->
<!-- keys: errata -->
<table width="80%"><tr><td>
<a name="2"><b>ZFS Errata #2</b></a>
<p>
<dl>
<dt><p><b>Type</b>
<dd>Compatibility
<dt><p><b>Severity</b>
<dd>Moderate
<dt><p><b>Description</b>
<dd>The ZFS packages were updated while an asynchronous destroy was in
progress and the pool contains an on-disk format incompatibility.
<dt><p><b>Automated Response</b>
<dd>No automated response will be taken.
<dt><p><b>Impact</b>
<dd>The pool cannot be imported until the issue is corrected.
<dt><p><b>Suggested Action for System Administrator</b>
<dd>
Affected pools must be reverted to the previous ZFS version where they can
be correctly imported.  Once imported all asynchronous destroy operations
must be allowed to complete.  The ZFS packages may then be updated and the
pool can be imported cleanly by the newer software.
<pre>
# zpool import
  pool: test
    id: 1165955789558693437
 state: ONLINE
status: Errata #2 detected.
action: The pool cannot be imported with this version of ZFS due to
	an active asynchronous destroy.  Revert to an earlier version
	and allow the destroy to complete before updating.
   see: https://zfsonlinux.org/msg/ZFS-8000-ER
config:

	test           ONLINE
	  raidz1-0     ONLINE
	    vdev0      ONLINE
	    vdev1      ONLINE
	    vdev2      ONLINE
	    vdev3      ONLINE
</pre>

Revert to previous ZFS version, import the pool, then wait for the
<b>freeing</b> property to drop to zero.  This indicates that all outstanding
asynchronous destroys have completed.

<pre>
# zpool get freeing
NAME  PROPERTY  VALUE    SOURCE
test  freeing   0        default
</pre>

The ZFS packages may be now be updated and the pool imported.  The on-disk
format incompatibility can now be corrected online as described in
<a href="#1">Errata #1</a>.
</pre>
</td></tr></table>
<br><br>
<hr>
<br><br>
<!-- end: templates/success -->
<!-- start: article -->
<!-- code: ZFS-8000-ER -->
<!-- keys: errata -->
<table width="80%"><tr><td>
<a name="3"><b>ZFS Errata #3</b></a>
<p>
<dl>
<dt><p><b>Type</b>
<dd>Compatibility
<dt><p><b>Severity</b>
<dd>Moderate
<dt><p><b>Description</b>
<dd>An encrypted dataset contains an on-disk format incompatibility.
<dt><p><b>Automated Response</b>
<dd>No automated response will be taken.
<dt><p><b>Impact</b>
<dd>
Encrypted datasets created before the ZFS packages were updated cannot be
mounted or opened for write. The errata impacts the ability of ZFS to correctly
perform raw sends, so this functionality has been disabled for these datasets.
</dd>
<dt><p><b>Suggested Action for System Administrator</b>
<dd>
System administrators with affected pools will need to recreate any encrypted
datasets created before the new version of ZFS was used. This can be
accomplished by using <b>zfs send</b> and <b>zfs receive</b>. Note, however,
that backups can NOT be done with a raw <b>zfs send -w</b>, since this would
preserve the on-disk incompatibility. Alternatively, system administrators can
use conventional tools to back up data to new encrypted datasets. The new
version of ZFS will prevent new data from being written to the impacted datasets,
but they can still be mounted read-only.
<pre>
# zpool status
  pool: test
    id: 1165955789558693437
 state: ONLINE
status: Errata #3 detected.
action: To correct the issue backup existing encrypted datasets to new
	encrypted datasets and destroy the old ones.
   see: https://zfsonlinux.org/msg/ZFS-8000-ER
config:

	test           ONLINE
	  raidz1-0     ONLINE
	    vdev0      ONLINE
	    vdev1      ONLINE
	    vdev2      ONLINE
	    vdev3      ONLINE
</pre>

Import the pool and backup any existing encrypted datasets to new datasets.
To ensure the new datasets are re-encrypted, be sure to receive them below an
encryption root or use <b>zfs receive -o encryption=on</b>, then destroy the
source dataset.
<pre>
# zfs send test/crypt1@snap1 | zfs receive -o encryption=on -o keyformat=passphrase -o keylocation=file:///path/to/keyfile test/newcrypt1
# zfs send -I test/crypt1@snap1 test/crypt1@snap5 | zfs receive test/newcrypt1
# zfs destroy -R test/crypt1
</pre>

New datasets can be mounted read-write and used normally. The errata will be
cleared upon reimporting the pool and the alert will only be shown again if
another dataset is found with the errata. To ensure that all datasets are
on the new version reimport the pool, load all keys, mount all encrypted
datasets, and check <b>zpool status</b>.

<pre>
# zpool export test
# zpool import test
# zfs load-key -a
Enter passphrase for 'test/crypt1':
1 / 1 key(s) successfully loaded
# zfs mount -a
# zpool status -x
all pools are healthy
</pre>
</td></tr></table>
<br><br>
<hr>
<br><br>
<!-- end: templates/success -->
<!-- start: article -->
<!-- code: ZFS-8000-ER -->
<!-- keys: errata -->
<table width="80%"><tr><td>
<a name="4"><b>ZFS Errata #4</b></a>
<p>
<dl>
<dt><p><b>Type</b>
<dd>Compatibility
<dt><p><b>Severity</b>
<dd>Moderate
<dt><p><b>Description</b>
<dd>An encrypted dataset contains an on-disk format incompatibility.
<dt><p><b>Automated Response</b>
<dd>No automated response will be taken.
<dt><p><b>Impact</b>
<dd>
Encrypted datasets created before the ZFS packages were updated cannot be
backed up via a raw send to an updated system. These datasets also cannot
receive additional snapshots. New encrypted datasets cannot be created
until the bookmark_v2 feature has been enabled.
</dd>
<dt><p><b>Suggested Action for System Administrator</b>
<dd>
First, system administrators with affected pools will need to enable the
<b>bookmark_v2</b> feature on their pools. Enabling this feature will
prevent this pool from being imported by previous versions of the ZFS
software after any new bookmarks are created (including read-only imports).
If the pool contains no encrypted datasets, this is the only step required.

If there are existing encrypted datasets, administrators will then need
to back these datasets up. This can be done in several ways. Non-raw
<b>zfs send</b> and <b>zfs receive</b> can be used as per usual, as can
traditional backup tools.

Raw receives of existing encrypted datasets and raw receives into
existing encrypted datasets are currently disabled because ZFS is not
able to guarantee that the stream and the existing dataset came from a
consistent source. This check can be disabled which will allow ZFS to
receive these streams anyway. Note that this can result in datasets
with data that cannot be accessed due to authentication errors if raw
and non-raw receives are mixed over the course of several incremental
backups. To disable this restriction, set the
<b>zfs_disable_ivset_guid_check</b> module parameter to 1. Streams
received this way (as well as any received before the upgrade) will
need to be manually checked by reading the data to ensure they are not
corrupted. Note that <b>zpool scrub</b> cannot be used for this purpose
because the scrub does not check the cryptographic authentication codes.
For more information on this issue, please refer to the zfs man page
section on <b>zfs receive</b> which describes the restrictions
on raw sends.

<pre>
# zpool status
  pool: test
 state: ONLINE
status: Errata #4 detected.
        Existing encrypted datasets contain an on-disk incompatibility
        which needs to be corrected.
action: To correct the issue enable the bookmark_v2 feature and backup
        any existing encrypted datasets to new encrypted datasets and
        destroy the old ones. If this pool does not contain any
        encrypted datasets, simply enable the bookmark_v2 feature.
   see: http://zfsonlinux.org/msg/ZFS-8000-ER
  scan: none requested
config:

        NAME           STATE     READ WRITE CKSUM
        test           ONLINE       0     0     0
          /root/vdev0  ONLINE       0     0     0

errors: No known data errors
</pre>

Import the pool and enable the <b>bookmark_v2</b> feature. Then backup any
existing encrypted datasets to new datasets. This can be done with
traditional tools or via <b>zfs send</b>. Raw sends will require that the
zfs_disable_ivset_guid_check is set to 1 on the receive side. Once this
is done, the original datasets should be destroyed.
<pre>
# zpool set feature@bookmark_v2=enabled test
# echo 1 > /sys/module/zfs/parameters/zfs_disable_ivset_guid_check
# zfs send -Rw test/crypt1@snap1 | zfs receive test/newcrypt1
# zfs send -I test/crypt1@snap1 test/crypt1@snap5 | zfs receive test/newcrypt1
# zfs destroy -R test/crypt1
# echo 0 > /sys/module/zfs/parameters/zfs_disable_ivset_guid_check
</pre>

The errata will be cleared upon reimporting the pool and the alert will
only be shown again if another dataset is found with the errata. To
check that all datasets are fixed, perform a <b>zfs list -t all</b>, and
check <b>zpool status</b> once it is completed.

<pre>
# zpool export test
# zpool import test
# zpool scrub # wait for completion
# zpool status -x
all pools are healthy
</pre>
</td></tr></table>
</body></html>
